feat: protobuf schema registry, typed storage, and persistence - #84
Merged
Conversation
adds the `protobuf` feature gate across the workspace. when enabled, ember-core gains prost-reflect for dynamic protobuf message handling. the feature cascades: ember-server/protobuf enables emberkv-core/protobuf and ember-persistence/protobuf.
SchemaRegistry stores compiled FileDescriptorSet descriptors and validates protobuf values at write time. supports registration, validation, introspection (describe/list), and recovery (restore). gated behind the `protobuf` feature flag.
adds a new Proto { type_name, data } variant to the Value enum, gated
behind the protobuf feature. updates PartialEq, type_name(), value_size(),
is_large_value(), and snapshot/recovery code to handle the new type.
type-checked read/write operations for protobuf values. no schema validation at this layer — that's the server's job. follows the same memory enforcement and expiry patterns as existing set/get.
adds ProtoSet/ProtoGet/ProtoType to ShardRequest/ShardResponse with dispatch arms calling the keyspace methods. AOF records ProtoSet and ProtoRegister handle persistence and recovery. also fills in missing read_payload_for_tag arms for tags 14-22 (HSET, HDEL, etc).
adds optional SharedSchemaRegistry to EngineConfig and Engine, with an accessor method for the server layer. implements Debug for SchemaRegistry manually since DescriptorPool doesn't derive it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
adds native protobuf value storage behind a
protobuffeature flag. this is PR 1 of 3 — the foundation layer that includes the schema registry,Value::Prototype, and full persistence support (AOF + snapshot + recovery). command parsing and server wiring come in PR 2.key additions:
crates/ember-core/src/schema.rs) — stores compiledFileDescriptorSetdescriptors usingprost-reflect, validates protobuf values at write time, supports registration, validation, describe, and restore (for recovery)Value::Protovariant — new value type carryingtype_name+data(Bytes), with memory trackingproto_set,proto_get,proto_typefollowing existing SET/GET patternsProtoSet,ProtoGet,ProtoTyperequest/response variants with dispatch armsTAG_PROTO_SET(23) andTAG_PROTO_REGISTER(24) record types with serialization/deserializationTYPE_PROTO(5) snap value type with read/write supportRecoveredValue::Protovariant, AOF replay for proto recordsschema_registryfield onEngineConfigandEnginewith accessorthe entire feature is gated behind
#[cfg(feature = "protobuf")]— compiles cleanly with and without the flag.what was tested
cargo test -p emberkv-core --features protobuf— 310 tests pass (9 new schema registry tests)cargo test -p emberkv-core— 301 tests pass (no regressions without feature)cargo test -p ember-persistence --features protobuf— 66 tests passcargo build -p ember-server --features protobuf— builds cleancargo build -p ember-server— builds clean without featuredesign considerations
Arc<RwLock<SchemaRegistry>>shared across connections.prost-reflectfor dynamic messages — no codegen required. users supply pre-compiledFileDescriptorSetbytes fromprotoc --descriptor_set_out.ember-server/protobufenablesemberkv-core/protobufwhich enablesember-persistence/protobuf, keeping dependency management clean.restore()is idempotent (skips duplicates silently) for safe AOF replay.register()returns an error on duplicates for user-facing commands.